feat(jobs): Add data retention jobs#4128
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
@TheodoreSpeaks let's consolidate the migrations into a single one, just delete the existing ones and run it once over all the changes in shcema.ts |
TheodoreSpeaks
force-pushed
the
feat/auto-redaction
branch
from
57194bd to
a3c1bab
Compare
TheodoreSpeaks
force-pushed
the
feat/auto-redaction
branch
from
a3c1bab to
63f7b59
Compare
TheodoreSpeaks
changed the title
|
@BugBot review |
PR SummaryHigh Risk Overview Adds an enterprise-only workspace settings surface (API + UI tab) to view effective retention (defaults vs configured) and to update per-workspace retention hours with auditing. Centralizes retention defaults and job dispatch in Reviewed by Cursor Bugbot for commit 18b3f24. Bugbot is set up for automated code reviews on this repo. Configure here. |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 4 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 18b3f24. Configure here.
| logger.error(`[${tableName}] Batch delete failed:`, { error }) | ||
| hasMore = false | ||
| } | ||
| } |
There was a problem hiding this comment.
DELETE without LIMIT makes batching logic ineffective
High Severity
The cleanupTable function in both cleanup-soft-deletes.ts and cleanup-tasks.ts uses BATCH_SIZE and MAX_BATCHES_PER_TABLE constants to suggest batched deletion, but the db.delete() call has no .limit() clause. PostgreSQL DELETE doesn't support LIMIT directly, so the query deletes all matching rows in a single unbounded operation. The check hasMore = deleted.length === BATCH_SIZE is effectively dead logic since the delete returns all rows, not a capped batch. Additionally, .returning({ id: sql\id` })loads all deleted row IDs into memory at once, risking OOM for large tables. Compare withcleanup-logs.ts, which correctly batches via SELECT with .limit(BATCH_SIZE)` followed by DELETE-by-IDs.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 18b3f24. Configure here.
| "version": "7", | ||
| "when": 1775952219230, | ||
| "tag": "0191_parched_living_mummy", | ||
| "breakpoints": true |
There was a problem hiding this comment.
Migration journal references non-existent SQL migration files
High Severity
The migration journal adds two entries — 0190_lean_terror and 0191_parched_living_mummy — but the corresponding .sql files don't exist in the migrations/ directory. Additionally, there are now two entries with idx: 190 (the pre-existing 0190_shocking_karma and the new 0190_lean_terror), creating a duplicate index. Running migrations will fail because the migration runner can't find the referenced files.
Reviewed by Cursor Bugbot for commit 18b3f24. Configure here.
| "type": "integer", | ||
| "primaryKey": false, | ||
| "notNull": false | ||
| }, |
There was a problem hiding this comment.
Snapshot defines task_redaction_hours column absent from schema
Medium Severity
The migration snapshot includes a task_redaction_hours column on the workspace table, but schema.ts does not define a corresponding taskRedactionHours field. The schema only adds logRetentionHours, softDeleteRetentionHours, and taskCleanupHours. This mismatch means the migration would create a DB column that's inaccessible from application code, and the next drizzle-kit generate would produce a migration to drop it.
Reviewed by Cursor Bugbot for commit 18b3f24. Configure here.
| runChildResults.reduce((s, r) => s + r.deleted, 0) + | ||
| runsResult.deleted + | ||
| chatsResult.deleted + | ||
| inboxResult.deleted |
There was a problem hiding this comment.
Feedback deletions excluded from total deleted count
Low Severity
The totalDeleted sum in runCleanupTasks includes runChildResults, runsResult, chatsResult, and inboxResult, but omits feedbackResult.deleted. The feedback deletion block (around line 218–253) successfully deletes copilotFeedback rows but those counts aren't reflected in the logged total.
Reviewed by Cursor Bugbot for commit 18b3f24. Configure here.
2 participants
Summary
Add data retention jobs. 3 jobs created:
Type of Change
Testing
Checklist
Screenshots/Videos